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(54) Title: OPERATION METHOD OF USER INTERFACE OF HAND-HELD DEVICE 




(57) Abstract: An operation method of a user interface of a handheld device, like a palm-size or pocket-size PC, mobile telephone, 
conununicator or equivalent, in which method data on a display of the device is changed for finding or selecting desired data, is 
characterized in that data (5a - 5e) on the display (2*) of the device is changed responsive to certain movements (F) of the device in a 
three-dimensional space of use thereof, and said movements of the device and the corresponding changes of the data on the display 
are defined so that a movement and a conresponding change of data on the display have a cause-consequence relationship which is 
^ analogous to cause-consequence relationships of the three-dimensional space of perception and action of a user (1). 
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Operation method of user interface of hand-held device 

The invention is related to an operation method of the user interface of a hand-held 
data-processing device. A hand-held device means here a palm-top or pocket computer, 
5 mobile phone, communicator or like device, essential features of which include displaying 
data on the display of the device and in which data on the display of the device is changed 
for finding and selecting a desired data. 

User interfaces of above mentioned devices are presently mostly of type "point and 
click". A general operation method of a user interface is browsing and selecting data by 
10 means of keys. Palm-top computers are usually managed with a touch sensitive display, a 
virtual keyboard, a pen, and often some auxiliary hard keys or buttons. Firstly, this kind of 
user interfaces are difficult to use for small size of the devices and correspondingly the keys 
or virtual keys. On the other hand, the use becomes more difRcuh as the devices are 
continuously provided with new applications and services which are to be used with the 
1 5 same small quantity of small-sized keys. Characteristric of these user interfaces, like user 
interfaces of computers and other data-processing devices, is generally also that they have 
their own artificial logics and rules which are fiilly adopted and known only by a few, 
normally technically orientated xisers. 

An object of the invention is to present such an operation method of the user interface 
20 of hand-held devices which to large extent removes many of the above mentioned problems. 
To reach this object, the operation method of the user interface of a hand-held device, 
like palm-top or pocket computer, mobUe phone, communicator or like, according to the 
invention, in which operation method data on the display of the device is changed for 
finding and selecting a desired data, is characterized in that what is defined by claim 1 . 
25 Other claims define variable embodiments of the invention. 

An advantage of the invention is that a user is able to easily adopt the operating rules of 
the user mterface because they are "natural". The use of the device is also ergonomically 
easier because there is no need to use small-sized keys or auxiliary devices, for example. 
The mvention and some embodiments thereof are described in further detail in the 
30 following with reference to the accompanying drawings, in which: 

Figs. 1 to 10 are schematic perspective views presenting examples of the operation 
method of the user interface of the invention; 

Figs. 1 1 to 13 are schematic fi-ont views of a device and data presented on the display 
thereof and present some further examples of the operation method of the user interface of 
35 the invention; 
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Fig. 14 is a schematic presentation of an arrangement of data and operation of an 
application of the user interface of the invention; 

Fig. 1 5 is a schematic perspective view of operation of a further application of the user 
interface of the invention; and 
5 Fig. 1 6 is a flow chart presenting a possible realization of the operation method of the 

user interface according to the invention. 

In Figs. 1 to 5, a user is holding a hand-held device 2 in his or her hand 3 and is using a 
device having a display 2\ The display 2' is a touch sensitive display, and the user is able to 
give a signal to the device by pressing the display with his or her thumb 4. Movements of 

10 the device 2 are detected by an acceleration measurement circuit installed in the device 
itself, for example, and the data obtained from the circuit is utilized in the operation of the 
user interface. Some possible ways of realizing the operation method of the user interface 
according to the invention are considered in further detail below. Figs. 1 to 5 present 
examples of the operation of the user interface. 

15 With reference to Fig. 1, pages 5a, 5b, 5c, 5d and 5e, which may be e.g. weather maps 

obtained as a resvdt of a web search, are browsed on the display 2' in response to moving the 
device 2 essentially perpendicularly to the display 2' thereof, as is indicated by arrow F. For 
the user, it is easy to think that he or she has a pile of pictures 5a to 5e ahead of him or her 
and that by moving in the pile he or she is able to see a picture at which he or she is at each 

20 time. In the same way, successive (or piled) pages of a book or other document may be 
thought to be browsed. The speed of browsing may be made dependent on quickness or 
intensity of the movement, i.e. on the magnitude of acceleration in the movement. The 
operation may also be arranged in such a way that a suitable small movement forward or 
backward and immediate stop bring the next page in the corresponding direction on the 

25 display. Furthermore, it may be arranged that a push of the touch sensitive display 2' gives a 
signal to the display control means in response to which signal changing of the data on the 
display is stopped^ This kind of virtual tise of depth increases significantly the capacity of a 
display having a restricted size. 

With reference to Fig. 2, an image on the display 2' is zoomed larger, image 6b, or 

30 smaller, image 6a, in response to moving the device 2 in the same way as above, essentially 
perpendicularly to the display 2' thereof, as is indicated by arrow Z. This operation method 
of the user interface is fully analogous to human action as he or she wants to examine details 
of an object, e.g. an image, more closely. Having foimd a desired magnification, changing of 
the data on the display may be stopped again by a push of the display 2\ for example. The 

35 functions of Figs. 1 and 2 may be used alternatively in a user interface according to 

^plication, whereby there is no indistinction of what is happening in response to moving 
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the device in this way. The function of Figs. 1 and 2 may also be combined in the operation 
of a user interface so that, for example, data is first browsed and then zoomed by means of 
the same movement. Then, it is advantageous to arrange a suitable selector for selecting one 
or the other of the functions. A selector of zooming may be a push of a certain area on the 
5 display, for example, whereby changing over from the zooming mode to the browsing mode 
may happen automatically by selection of a desired magnification ratio. 

In Fig. 3, reference sign 7a indicates image data which is significantly larger than the 
display. The capacity of the display of a hand-held device is not large enough for large 
entities of factual or image data. In response to lateral movement of the device in essentially 
10 the plane of the display 2' , as is indicated by arrows N, E, S and W, data 7b on the display is 
changed as if a window were moved above a larger image formed by the image data 7a. 
Having foimd a desired place, changing of data 7b on the display may again be stopped by a 
push of the display. It is logical to accomplish the functions of Figs. 2 and 3 together 
whereby the image data may be examined more thoroughly after zooming by selecting the 
1 5 data in the way described by Fig. 3. Moreover, e.g. nxmiber selection may be realized in this 
way without keys by moving the device above a large virtual keyboard and selecting the 
desired numbers one by one. This function makes it possible to read full-length web pages 
with a palm-top computer, for example. 

With reference to Fig. 4, in response to a quick rotational movement of the device 2 as 
20 if around an axis formed essentially by the edge 9 thereof, i.e. in response to swinging the 
device like turning page of a book, as is indicated by arrow P, data 8a, 8b, 8c on the display 
2' is changed correspondingly, e.g. a page of an electronic book is turned. By the direction 
of the swing, the direction of turning p^e is selected. The function may be arranged so that 
a small swing wdth quick stopping tums one page in the direction of the swing, and that in 
25 response to a more intensive swing, pages are browsed more quickly until the browsing is 
stopped by a push of the display. 

In Fig. 5, an application is illustrated in which any part 9b of image data 9a forming a 
panorama picture covering the whole sphere of perceptual space (or respectively of image 
data forming a cylindrical picture) may be examined at a time. In the operation method of a 
30 user interface according to the invention, in response to moving the device 2 as if moving a 
picture on a corresponding spherical surface or in response to changing orientation of the 
device, that part 9b of the image data 9a which corresponds to the orientation of the display 
is obtained on the display 2\ At the same time, stopping and zooming of the image may be 
applied in the way described above. It may be contemplated also that a panorama picture is 
35 rotated by giving a push with a movement of the device to a desired direction, whereby the 
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panorama picture is correspondingly kept moving until moving is stopped by a push of the 
display, for example. 

Figs. 6 to 10 present embodiments of the user interface of the invention in which data 
on the display is changed by tilting the device. In Fig. 6, an example is presented in which 
data, illustrated by data Dl, on the display 2' of the device 2 is scrolled in the way indicated 
by arrow Rl to the direction to which the device is tilted in the way indicated by arrow Tl . 
The initial position of the device is indicated by broken line and the tilted position by solid 
line. In other words, in response to tilting the device, the data on the display is rolling or 
running to the direction to which the device is tilted, which is fully analogous to cause- 
consequence realtionships of the real world. This may be realized in such a way, for 
example, that the changing velocity is the grater the more the device is being tilted or is 
tihed from the initial position. Rolling or running of the data may be stopped by returning 
the device to the initial position or by means of a key on the device, for example. 

Fig. 7 presents an embodiment suitable for moving and searching objects on a map, for 
example. On the display 2,' a cursor C is placed which keeps its place thereon, and tilting 
the device in the way indicated by arrow T2 causes that data, illustrated by data D2, starts 
moving from the tilting direction towards to the cursor C in the way indicated by arrow M2. 
In other words, the cursor is moving on the data fomiing a map, for example, to the direction 
to which the device is tilted. 

In the example of Fig. 8, data on the display 2' of the device 2 is illustrated by data D3. 
As the user is tilting the device 2 from the position indicated by broken line away from 
himself or herself, like going closer to the data on the display, the data on the display is 
enlarged, which is illustrated by a graph indicating enlarging of data D3 and arrow Z2. In the 
example of Fig. 9, data D4 on the display is correspondingly reduced, as is indicated by a 
graph illustrating the reduction and drawing away and by arrow Z3, by tilting the device 
towards the user in the way indicated by arrow T4. 

The embodiments of Figs. 6, 7, 8 and 9 may be combined in the user interface of a 
device in such a way, for example, that there is a key on the device for selecting the 
operation mode (scrolling or zooming) whereby, first, an object is searched by tilting in the 
scrolling mode, for example, and having found the object, changing over to the zooming 
mode is made whereby tilting results in zooming the data on the display. Scrolling by tilting 
may, of course, be combined with zooming in the way described with reference to Fig. 2. 

Fig. 10 presents schematically an embodiment of the user interface of the invention for 
selecting data objects on the display. Data objects D5 to DIG may be alternatives in a menu 
or selection buttons on a page, for example. In the initial position of the device 2 indicated 
by broken line, data object D5 selected which is indicated by hatching in the figure. Tilting 
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the device in the way indicated by arrow T5, the selection is moved to the direction of 
tilting, as is indicated by arrow M3, and is here moved to data object D9. Having selected 
the desired data object, it may be locked by a key on the device or by a return movement of 
the device, for example. For the sake of simplicity, tilting the device to only one direction 
and selection from data objects located one below the other are presented here, but in the 
same way it is, of course, possible to move between selectable objects also laterally. 

Fig. 1 1 presents a further example of the embodiment described with reference to Fig. 
3. Wide data Dl 1 consists of objects which are described by letters arranged to form a 
matrix. By moving the device in the way indicated by arrows N, E, S and W one is as if 
moving above the data and looking at it through a window formed by the display 2'. As the 
desired data, here letter q, is found, it may be selected by means of a key on the device, for 
example, A comparable function may be realized also so that, instead of moving the device 
laterally, it is tilted to a direction to which one desires to move on the data. 

Fig. 12 presents a further embodiment of the user interface of the invention for 
selecting an object on the display. On the display 2' of the device 2 there are objects Ol to 
09 from which the selection is made. In the initial state at left, the cursor is close to the 
upper right comer of the display. In this embodiment, cursor C is as if anchored in place in 
the real world. As one desires to select object 07, the device 2 is moved laterally as if 
moving a picture xmder the cursor so that the desired object 07 is con:iing under the cursor C 
at right in the figure. There are keys 21, 22 and 23 on the device the operation of which may 
correspond to the operation of the buttons of a mouse. In fact, the embodiments of Figs. 10 
to 12 are solutions which in hand-held devices replace solutions, like a mouse, in 
conventional computers which are needed for moving, moving a cursor or making selections 
on the display. 

Fig. 13 presents an embodiment in which data objects, pages, cards or like, CI, ... , Ci, 
... , Cn are arranged in a stack or one above the other on the display 2' of the device, like 
e.g. pages of a book in the real world. The data objects may be cards presenting articles of 
commerce or web pages, for example. The objects are browsed in the way indicated by 
arrow M4 forward and backword by tilting the device correspondingly either forward in the 
way indicated by arrow T6 or backward in the way indicated by arrow T7. By means of keys 
24 to 26, objects may be selected and files, programs, etc. behind them be opened. There 
may be several modes in the user interface of a device, like browsmg, scrolling, selecting 
and zooming described above, and e.g. key 24 may be for selection of mode and other keys 
25, 26 and 27 may operate like the buttons of a conventional mouse. 

Fig. 14 presents an application of the user interface of the invention in which data 
objects are arranged in radial stacks S 11 to SI 8 to extend outwards from the circle of the 
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centre in which a device 2 and a user are. The first page of each stack, e.g. P 12 or PI 6, 
defines data therebehind. Front pages may be browsed by turning the device laterally in the 
way indicated by arrow T8, for example, so that direction A12 and front page P 12 are 
selected. The stack of data objects behind it may be browsed either by moving or tilting the 
5 device 2 forward and backward in the way indicated by arrow F 1 . In this way. e.g. a user 
interface for electronic shopping may be arranged and make perceiving the places of the 
products and orientation in a virtual shop easier. 

Fig. 15 presents a further developed application of the embodiment described with 
reference to Fig. 5. Circles Rl and R2 present now image data forming a panorama picture 

10 or a 3D picture, in general, covering the whole sphere of the perception space. The image 
data may be also image data obtained fi^om a video camera connected to the device. Three- 
dimensional image data D12 describing a thing, for example, may be brought to this image 
data and set in a desired place and desired position in "the real surroundings". In response to 
moving the device 2 in the way described with reference to Fig. 5, e.g. as indicated by arrow 

15 T9, both the background data and 3D data D\2\ DI2'* describing the thing are changed in 
accordance with the positions of the device to correspond to viewing directions A 1 and A2, 
so that the thing may be examined from different directions in "the real surroundings" 
thereof. This may be applied e.g. by searching the 3D model of a sofa fi-om a web shop and 
by placing it in a desired place in an image data presenting one's house whereby it may be 

20 examined from different directions to see how the sofa looks like in the real surroundings 
thereof. 

A solution according to the invention may be realized by providing a hand-held device 
by a multi-axial accelerometer, for example, and with suitable circuits and programs co- 
operating vsdth the operating system of the device and possibly application programs for 

25 processing and interpreting measurement results so that a change of data on the display 
corresponding to a movement detected by the accelerometer is carried out. E.g. the 
realization with measurement of accelerations is based on application of as such known 
technical solutions, and a person skilled in the art, having provided with instructions and 
specifications, is able to realize the operation of the user interface according to the invention 

30 with reasonable efforts. 

In small-sized data-processing devices, like palm-top computers, with prior art 
hardware technology, it is not possible to realize operation systems and applications with a 
capacity and usability which were even close to the level reached by larger data-processing 
devices, like desktop or laptop computers. It is also impossible to include conventional 

35 storage means, like hard disk, floppy disk or CD-ROM drivers, in hand-held devices. A 

solution of these problems may be a two-part data-processing device wherein in a hand-hald 
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part there is only a part of necessary circuits and programs in addition to a display. The most 
of the circuits and programs are in another portable part held by the user, a wireless link 
connecting this part to the hand-held part. The wireless link may operate with IR or radio 
frequencies. In this kind of device, it is easy to realize at the same time a system which 

5 detects movement or position of the hand-held part in relation to the portable part. The 
operation method of the user interface according to the invention may then be realized by 
means of this system. 

In the following, a further embodiment of the invention based on detecting 
accelerations is considered with reference to Fig. 16. The device includes a multi-axial 

10 accelerometer and necessary circuits and programs for measurements. The device being 
switched on, it is monitoring accelerations and in phase 1 1 is interpreting that an 
acceleration above a certain threshold is possibly an initial stage of a movement defined in 
the operation method of the \iser interface. In response to this, timers TDl and TD2 are 
started in phase 12. TDl sets a very short experimentally determined time of the order of 

15 miUiseconds from the detected start of the movement, in which the actual direction and 

magnitude of the acceleration specify the movement which the user gives to the device. TD2 
sets a longer time expiring of which stops changing data on the display if no other causes for 
stopping appeared. At the time of expiration of time TDl, the prevailing acceleration vector 
is measured in phase 13. In phase 14, it is examined if a detected vector is a defined vector, 

20 i.e. if it is corresponding to any movement defined in the operation method of the user 
interface. If not, the operation returns back to the beginning 10 and phase 1 1 to monitor 
accelerations of the device. On the other hand, if a vector is a defined vector, data on the 
display is changed in phase IS, e.g. is browsed, zoomed, etc., according to a corresponding 
algorithm. As is described above, e.g. detemiination of the browsing speed of data on the 

25 display on the basis of the magnitude of an acceleration vector may be related to this. In 
phase 16, it is also monitored at the same time if time TD2 is expired. If so, the operation is 
forwarded to phase 19 in which changing of data on the display is stopped and the procedure 
is finished in phase 20. If time TD2 is not expired, it is monitored if any opposite vector to 
the detected defined vector or any other stopping signal, like a push of the display, appear in 

30 phase 1 7. If an opposite vector or a stopping signal are detected in phase 1 8, changing data 
on the display is stopped in phase 19 and the procedure is finished. If no opposite vector or 
other stopping signals are detected in phase 1 8, the operation returns back to monitor above 
mentioned issues. 

The operation method of the user interface of the invention may be realized also in 
35 other ways than by utilizing acceleration measurements. Any technique by which changes of 
place and position of the device may be meastired may be applied. A solution which may be 
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contemplated is technique in which a place and orientation of the device are detected by 
sending pulsed DC magnetic field and by measuring it with a detector in the device. With 
this technique, both a place and position of the device in three-dimensional space may be 
fiiUy detected. 

5 A possible solution is also the technique on which so called optical mouse is based and 

in which a movement and speed thereof are detected by observing and analyzing any surface 
in the vicinity, which in this case could be a surface on the user, for example. By combining 
this with distance measurement, which is easy to realize, and by analyzing changes of 
distance and relative movement, changes of place, movement and position of a device may 

10 be detected. 

The invention may, of course, be realized in various ways. It may be contemplated, for 
example, that for making detection of a defined movement more reliable, an acceleration is 
measured many times within a short time window, and a defined movement is detected if 
any one of the measured vectors meets the requirements for detection. Naturally, also a more 

15 complicated analysis of a movement based on successive measured acceleration vectors may 
be contemplaed. A movement of a device may be detected also with an arrangement, for 
example, in which distances are measured between transmitters and receivers placed in the 
device, on one hand, and on the user, on the other hand. 

A user interface according to the invention may be realized according to use and 

20 necessary features of a hand-held device as a suitable combination of the embodiments 
presented above, the user interface including several operation modes in which the same 
movement of the device may correspond to different changes of data on the display. As is 
stated above in relation to some embodiments, there may be a key, for example, on the 
device for selecting different operation modes. 

25 The invention may be varied within the scope of the accompanying claims. 
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Claims 



1 . An operation method of the user interface of a hand-held device, like palm-top or pocket 
computer, mobile phone, communicator or like, in which operation method data on the 

5 display of the device is changed for finding and selecting a desired data, characterized in 
that: 

data (5a - 5e; 6a, 6b; 7b; 8a. 8c; 9b; Dl; D2; D3; D4; D5 - DIO; Dll; CI - 
Cn; Sll - S18) on the display (2*) of the device is changed in response to defined 
movements of the device (F; Z; N, E, S, W; P; Tl, T2, T3, T4, T5; T6, T7; T8, Fl; Fig. 5) in 
10 the three-dimensional space of use thereof; and 

said movements of the device and the corresponding changes of data on the 
display are defined so that a movement and a corresponding change of data on the display 
have a cause-consequence relationship which is analogous to cause-consequence 
relationships of the three-dimensional space of perception and action of a user (1). 

15 

2. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to a movement (F; T6, T7) of the device perpendicxilarly 
to the display (2') thereof data objects (5a, 5b, 5c, 5d, 5e; SI, CI, Ci, Cn) are 
browsed on the display dependent on the direction of the movement either forward or 

20 backward, respectively. 

3. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to moving (F) the device forward or backward 
perpendicularly to the display (2') thereof data objects (5a, 5b, 5c, 5d, 5e) are browsed on the 

25 display dependent on the direction of the movement either forward or backward, 
respectively. 

4. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to tilting (T6, T7) the device (2) data objects (SI, CI, 

30 Ci, . . Cn) placed one above the other are browsed (M4) on the display (2') to a direction 
conesponding to the durection of tilting. 

5. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to moving (Z) the device (2) perpendicularly to the display 

35 (2') thereof towards the front of the display data (6a) on the display is zoomed larger (6b). 
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6. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to moving (Z) the device (2) perpendicularly to the display 
(2') thereof towards the back of the display data on the display is zoomed smaller. 



5 7. An operation method of the user interface of a hand-held device according to claim 1 , 
characterized in that in response to moving (N, S, W) the device (2) essentially in the 
direction of the plane of the display (2') thereof it is moved correspondingly (7b) in a 
displayed data (7a) which forms an image larger than the display. 

10 8. An operation method of the user interface of a hand-held device according to claim 1 , 
characterized in that in response to moving (N, E, S, W) the device (2) essentially in the 
direction of the plane of the display (2*) thereof data (q) from data (Dl 1) larger than the 
display is selected. 

15 9. An operation method of the user interface of a hand-held device according to claim 1 , 
characterized in that a cursor (C) is placed on the display (2') and in response to moving 
(N, E, S, W) the device (2) essentially in the direction of the plane of the display (2') thereof 
the cursor (C) is moved against the movement for selecting data (Ol - 09) on the display as 
if by moving data on the display in the real world below the cursor keeping its place in the 

20 real world (Fig. 12). 

10. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to swinging (P) the device (2) like turning page also data 
(8a, 8b, 8c) on the display (2') is changed in the way corresponding to turning page. 

25 

1 1 . An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to tilting (Tl) the device (2) data (Dl) on the display (2') 
is rolled (Ml) to the direction of tilting. 

30 12. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to tilting (T2) the device (2) data (D2) is moved on the 
display (2') from the direction of tilting towards a cursor (C). 

13. An operation method of the user interface of a hand-held device according to claim 1, 
35 characterized in that in response to tilting (T3) the device (2) away from a user data (D3) 
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on the display (2') is enlarged (Z2) and / or in response to tilting (T4) the device (2) towards 
a user data (D4) (2') is reduced (Z3). 

14. An operation method of the user interface of a hand-held device according to claim 1, 

5 characterized in that in response to tilting (T4) the device (2) selection of data (D5) on the 
display (2') is moved (M3, D9) into the direction of tilting. 

15. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that data is arranged virtually in stacks (SI 1 - SI 8) on a circle surrounding 

10 a user (a device (2)) whereby selection of a stack (S 12) is carried out by turning (T8) the 

device and the stack (SI 2) is browswd by moving or tilting (Fl) the device into the direction 
of the stack (SI 2) and away from it, respectively. 

16. An operation method of the user interface of a hand-held device according to claim 1, 

15 characterized in that in an application (Fig. 5), in which displayed data is a desired part of a 
panorama picture (9a), in response to turning the device (2) displayed data (9b) is changed 
so that the change of the viewing direction and the displayed part of the panorama picture 
(9a) correspond to turning. 

20 1 7, An operation method of the user interface of a hand-held device according to claim 1 , 
characterized in that in an application(Fig. 1 5), in which displayed data is a desired part of 
a panorama picture or 3D image data (Rl, R2) 3D, data (D12) describing a thing is placed in 
the image data and in response to turning the device (2) both the panorama picture data or 
3D image data (Rl, R2) and 3D image data (D12\ D12'') are changed on the display (2') to 

25 correspond to the viewing direction (Al, A2) corresponding to the orientation of the device 
(2). 

18. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that in response to stopping or significant slowing (18) of a movement 

30 changing of data on the display is stopped. 

19. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that changing of data on the display is stopped by giving a signal to the 
device in other way than by means of movement of the device. 
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